Papers with fully automated framework
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction (2025.findings-acl)
Copied to clipboard
Yuxin Jiang, Yufei Wang, Chuhan Wu, Xinyi Dai, Yan Xu, Weinan Gan, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
| Challenge: | Existing methods for generating and curating high-quality instruction-tuning data rely heavily on the quality of seed data or strong assumptions about the structure and content of web documents. |
| Approach: | They propose a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. |
| Outcome: | The proposed framework outperforms state-of-the-art baselines by 16.65% across four instruction-following benchmarks. |
Lexical Semantic Change Discovery (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Lexical Semantic Change Detection are limited. |
| Approach: | They propose a shift from change detection to change discovery by fine-tuning a type-based and a token-based approach on recently published German data. |
| Outcome: | The proposed models can be applied to discover new words undergoing meaning change from the full corpus vocabulary. |
Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to red teaming focus on searching for individual adversarial inputs. |
| Approach: | They propose a framework for automated adversarial data generation that inverts harmless constitution into constitution of toxicity and iteratively refining model outputs through critique–revision pipeline. |
| Outcome: | The proposed framework generates diverse, high-quality toxic data without human annotation and significantly improves semantic coherence without sacrificing adversarial strength. |
PromptWizard: Optimizing Prompts via Task-Aware, Feedback-Driven Self-Evolution (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. |
| Approach: | They propose a framework for discrete prompt optimization that generates human-readable prompts using feedback-driven critique and synthesis process. |
| Outcome: | The proposed framework improves prompt quality across 45 tasks and reduces API calls, token usage and overall cost. |
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models (2026.acl-long)
Copied to clipboard
Yanbin Yin, Kun Zhou, Zhen Wang, Xiangdong Zhang, Yifei Shao, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin, Zhiting Hu
| Challenge: | closed-ended question-based benchmarks struggle with saturation as newer models emerge . crowd-sourced leaderboards rely on costly and slow human judges . |
| Approach: | They propose a framework that leverages collective intelligence from all large language models to evaluate each other. |
| Outcome: | a new framework enables a democratic, pairwise evaluation of all large language models . it achieves 97% correlation with human judgements, while significantly reducing the cost. |
Identifying the Achilles’ Heel: An Iterative Method for Uncovering Factual Errors in Large Language Models (2026.findings-acl)
Copied to clipboard
Wenxuan Wang, Yuk-Kit Chan, Zixuan Ling, Shi Juluan, Youliang Yuan, Jen-tse Huang, Yifei Zhang, Wenxiang Jiao, Zhaopeng Tu, Michael R. Lyu
| Challenge: | Current methods for evaluating LLMs’ veracity are limited by the need for extensive human labor, test data contamination, or limited scope, hindering efficient and effective exposure of errors. |
| Approach: | They propose a framework that extracts fact triplets to generate diverse question types using rule-based natural language processing techniques. |
| Outcome: | The proposed framework can trigger factual errors in up to 55% of questions in large LLMs while maintaining coverage of questions. |
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks suffer from semantic drift and context loss, which can lead to misleading performance metrics. |
| Approach: | They propose a fully automated framework to enable translation of large language models . they propose to use universal self-improvement and multi-round ranking methods to improve translation quality . |
| Outcome: | The proposed framework surpasses existing benchmarks in eight languages and improves translation quality across multilingual domains. |